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We explore the phase diagram of the strongly correlated Hubbard model with intrinsic spin orbit 
coupling on the honeycomb lattice. We obtain the low energy effective model describing the spin 
degree of freedom. We study the resulting model within the Schwinger boson and Schwinger fermion 
approaches. The Schwinger boson approach gives the boundary between the spin liquid phase and 
the magnetically ordered phases, Neel order and incommensurate Neel order. We find that increasing 
the strength of the spin orbit coupling, narrows the width of the spin liquid region. The Schwinger 
fermion approach sheds further light on the nature of the spin liquid phase. We obtain three 
different candidates for the spin liquid phase within the mean field approximation which are gapless 
spin liquid, topological Mott insulator, and the chiral spin liquid phases. We argue that the gauge 
fluctuations and the instanton effect may suppress the first two spin liquids, while the chiral spin 
liquid is stable against gauge fluctuations due to its nontrivial topology. 

PACS numbers: 71.10.Fd,71.10.Pm,03.65.Vf 



I. INTRODUCTION 

The spin-orbit interaction driven topological insula- 
tor(TI) phase of matter with gapped bulk and protected 
gapless edge excitations has spurred a renewed inter- 
est in the topological states of matter. Kane and Mele 
proposed the first and the simplest model exhibiting TI 
phase for noninteracting electrons on the two dimensional 
honeycomb latticc""^'^. The topological nature of the TI 
phase and its Z2 structure due to the time reversal in- 
variance (TRI), protects the edge excitation from An- 
derson localization^'^. The absence of localization means 
these systems should be experimentally realizable as they 
are^'^. One crucial question that naturally arises is what 
happens to the TI starting from interacting electron sys- 
tems. The simplest interaction term is the onsite Hub- 
bard repulsion that causes strong correlation between 
electrons. Recently, Rachel and Le Hur in Ref.^ have 
studied the phase diagram of the Kane-Mele model in the 
presence of onsite Hubbard interaction. They concluded 
that the TI phase is stable against onsite repulsion up to 
a critical value of the interaction strength. Their result 
has been verified by several other authors both numeri- 
cally and analytically®"^^. 

The phase diagram of the Hubbard model has been 
extensively studied and various techniques support the 
existence of a spin liquid phase proximate to the Mott 
metal-insulator transition point^^"^^. In this paper, we 
are going to address the issue of the possibility of ob- 
serving the spin liquid phase in the phase diagram of 
the Kane-Mele-Hubbard model. Spin liquids may oc- 
cur when the charge degree of freedom is gapped and 
as a result frozen due to the interaction (Mott insula- 
tor), though the spin degree of freedom can be either 
gapless or gapped. Consequently, we can integrate out 
the charge degree of freedom as it is gapped with a gap 
of the order of onsite repulsion in the strong correlation 



regime. The resulting model describes the spin degree 
of freedom only. In the pure Hubbard model, that pro- 
cedure leads to the derivation of the extended Jl — J2 
Heisenberg model. Adding spin-orbit interaction to the 
Hubbard model, through the Kane-Mele term, introduces 
a new term for the second neighbor that is of the form 
92{-Sx{i)Sa;{j) -Sy{i)Sy{j) + S^{i)S:,{j)), where 52 = 
4AgQ/C/^. Combining this term with the Heisenberg 
interaction for the next nearest neighbor (NNN) yields 
an anisotropic XXZ model for the second nearest neigh- 
bor, while the nearest neighbor Heisenberg interaction 
remains intact. In this paper, we study the rich phase 
diagram of this model. Using a combination of the mean 
field results, gauge theory, instanton effect and topolog- 
ical arguments, we demonstrate that the phase diagram 
hosts a region of chiral gapped spin liquid phase^^ up to 
a critical value of the 92/ J2 and for large enough J2/J1. 
An interesting possibility is the emergence of the gapped 
topological spin liquid phase (topological Mott insulator) 
which is the same as TI phase, except that its charge de- 
gree of freedom is gapped. For large enough values of the 
92/J2i we show that the in plane XY magnetic ordering 
wins over the topological spin liquids after taking instan- 
ton effect into consideration. For small values of J2/J1 
and 92! J\i we obtain a gapless spin liquid which is shown 
to be unstable toward Neel order or valence bond solid 
(VBS) state. 

This paper is organized as follows: in section II, we 
introduce our model that describes the effective action 
for the spin degree of freedom in the strongly correlated 
Kane-Mele-Hubbard model and we derive Kane-Mele- 
Heisenberg (KMH) model. Section HI aims at studying 
KMH model using Schwinger boson approach. Within 
the mean field approximation, we investigate the possi- 
bility of the spin liquid phase in KMH model. Our study 
yields a phase diagram with 1- Neel order 2- incommen- 
surate Neel order 3- gapped spin liquid phase within the 
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mean field level. Then we focus on the spin liquid phase 
and study it further in section IV using the Schwinger 
fermion approach. Within the mean field approximation 
we study the competition between three different kinds 
of spin liquids. In section V, we discuss the gauge theory 
of the Schwinger fermion mean field states. We argue 
that the microscopic KMH enjoys an SU(2) gauge degree 
of freedom. However the SU(2) group may break down 
to its U(l) or Zi subgroup in the mean field state af- 
ter Anderson- Higgz mechanism. Moreover, a mean field 
state with nonzero Hall conductance can suppress gauge 
fluctuations further. We also take the effect of instan- 
tons into consideration to have a more accurate account- 
ing of gauge fluctuations. Using topological arguments 
and gauge theory, in section VI, we argue that among 
the proposed spin liquids, the chiral gapped spin liquid 
with nonzero Hall conductance for Schwinger fermions is 
stable against gauge fluctuations, while two other states 
may undergo transition to spontaneously broken symme- 
try phases. 




FIG. 1. (Color online) Kane-Mele model for the topological 
insulators. Arrows denote the phase of the hopping to the 
NNN for the spin up electrons. The phase for the spin downs 
is opposite to that of spin up electrons. Assigning a phase ^o- 
to the hopping of spin a electrons induces a 3(/>ct magnetic flux 
for spin ct electrons moving around the depicted triangles. It 
is this nonzero spin dependent fiux that causes spin quantum 
Hall effect for non-interacting electrons. 



II. MODEL 

The Kane-Mele-Hubbard model on the honeycomb lat- 
tice is described as 



Integrating out the Kane-Mele term leads to the fol- 
lowing effective Hamiltonian 



n 



((■'J)>,' 



(1) 

where i, U, and Aso s-re the nearest neighbor hop- 
ping energy, the strength of the on-site repulsion, and 
the second-neighbor spin-orbit coupling strength, respec- 
tively. Here Cia (cL) annihilates (creates) an electron 
with spin a on site i. Vij is introduced so as to obtain a 
nonzero flux turning around any triangular path and is 



defined as m, „■ — -r^ — J-r.z where c?, and c?,- are two short- 



est vectors that connect sites i and j i.e. di + dj = Ri — Rj 
(see Fig. [1]). Since we are interested in studying the in- 
terplay between the strong correlation and the topology 
of the band structure, we only consider the intermediate 
and the large U/t and U/Xgo limit of the Kane-Mele- 
Hubbard model. 

For the parameter space defined above and at half fill- 
ing (undoped case) , we can use standard techniques such 
as canonical transformations or the second order pertur- 
bation to obtain an effective Hamiltonian for the spin 
degree of freedom. Integrating out the hopping to the 
nearest neighbor term, we obtain the Jl — J2 Heisenberg 
model 



J2 



(2) 



where Ji and J2 are related to the parameters of the 
Hubbard model as follows 



Jl 
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where 32 — 4-^. This term generates a ferromagnetic 
interaction for the x and y components and antiferro- 
magnetic interaction for the z component of the spin op- 
erator. Considering the antiferromagnetic Heisenberg in- 
teraction, 52 adds frustration to the S^S^ interaction and 
reduces frustration from the in plane S^S^ -I- S^S^ inter- 
action. Therefore, when 172 term dominates over the J2 
Heisenberg interaction, it tends to align spins in the XY 
plane. 

The effective Hamiltonian is the simi of the Hj^^j.^ 
and Hg^ as follows 



H — Jl ^ S^.Sj 



((ij)) 



where 
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(6) 



In order to obtain the phase diagram of the strongly 
correlated Kane-Mele-Hubbard model, we can invert the 
above equations to solve U/t and Xso/t in terms of Ji — 
J2,± — J2.ll parameters as follows 




(J2.ll - J; 



'2,_L 



2(Ji +2J2.11 +2J2,_l) 
Jl + 2J2^|| + 2J^ 

J2,\\ + J2,_L 



(7) 
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Since the above Hamiltonian describes only the spin 
degree of freedom and is a generahzation for the Heisen- 
berg model, we name it Kane-Mele-Heisenberg Hamilto- 
nian. Because of the extra parameter in this model, its 
phase diagram is expected to be much richer than the 
phase diagram of the Ji — J2 Heisenberg model which 
is extensively studied in the literature^*"^^'^'^'^"'^. In this 
paper, we study the phase diagram of the Kane-Mele- 
Heisenberg Hamiltonian using several theoretical and nu- 
merical approaches. Before presenting formal discussions 
we would like to comment on the possible phases for the 
above Hamiltonian based on general arguments. The 
phase diagram depends on the two ratios, pi = J\\/Ji 
and p2 = Jl/J||. Naively speaking, the next nearest 
neighbor interaction becomes important when pi is com- 
parable to one. When pi is small, we expect a gapless 
spin liquid phase within the mean field approximation. 
On the other hand, for large values of pi, when P2 — > 1 
the model is closer to the Ji — J2 Heisenberg model and 
we can neglect the spin orbit coupling term. For those 
parameters, we expect a chiral spin liquid phase with 
gapped spin excitation spectrum which is described by 
the Haldane model in the mean field approximation. In 
the opposite limit where p2 — > —1, the Heisenberg in- 
teraction is negligible and the spin orbit coupling dom- 
inates. In this regime, a topological spin liquid phase 
with nonzero spin Hall conductance is expected which is 
described by the Kane-Mele model within the mean field 
level. In the remainder of this paper we present calcula- 
tions based on the Schwinger boson/fermion approaches 
to study the KMH model in more details. 



III. SCHWINGER BOSONS APPROACH 

Heisenberg interaction enjoys a global SU(2) spin rota- 
tion symmetry. Forgetting about the quantum mechani- 
cal nature of the spin operators, a simple classical anal- 
ysis of the spontaneous symmetry breaking yields the 
anti-ferromagnetic Neel ordering as the groundstate of 
the Heisenberg model. However, taking c^uantum fluc- 
tuations may melt the Neel order with solid-like long 
range order(LRO) down to a spin liquid which lacks LRO 
and symmetry breaking down to the lowest tempera- 
tures. One way to study this quantum transition is the 
Shwinger boson approach. Spin operators can be written 
in terms of Schwinger bosons (slave bosons or partons). 
Bosons exhibit two different behaviors. Their excitation 
energy is either gapless or gapped. In the gapless excita- 
tion case, Shwinger bosons condense at some momentum 
vector through the Bose-Einstein condensation (BEG). 
Following the definition of the spin operators in terms 
of these new Schwinger boson quasiparticles, they ac- 
quire a nonvanishing expectation value i.e. magnetic or- 
der emerges in the ground-state. The spatial pattern of 
the magnetic is determined by the momentum at which 
Shwinger bosons condense. 

the second case where Shwinger bosons are gapped and 



therefore do not undergo BEG transition even at zero 
temperature, leads to a vanishing expectation value for 
the spin operators. Subsequently, we obtain a spin liquid 
phase by definition in that case. 

What if we add the spin interaction caused by the 
spin-orbit interaction? According to the equation [4], 
the (72 term breaks the global SU(2) spin rotation down 
to the global U(l) spin rotation around the z-axis. This 
means within the classical physics, the spin operator has 
a nonzero in-plane value (XY ordering) , however there is 
still a degree of freedom for choosing the direction of that 
axis in the XY plane. Again, the quantum mechanics can 
change the story drastically. Schwinger boson approach 
can help us to overcome this dilemma in determining the 
fate of the spin ground-state. In the following, we present 
more detail on applying the Schwinger fermion approach 
and extract the phase diagram. 

Before staring with the Schwinger boson model, we 
would like to comment on the idea behind that proce- 
dure. To that end, we first employ the hardcore boson 
representation of spin operators, in which an empty site 
denotes the spin down and an occupied site denotes the 
spin up states at that site. Therefore, we use the follow- 
ing mappings 



S+^dl , S-^d. , S^^djd,--, (8) 

where di is the annihilation operator of hardcore boson 
at site i. It is easy to check that the above definitions 
recovers the SU(2) group symmetric relations for the 
spin generator. The dimension of the Hilbert is also two 
per site as we used hardcore bosons. However, working 
with hardcore bosons is hard. An acclaimed method is 
the slave boson where employs two types of bosons, d]^ 
which denotes the creation of the hardcore boson and 
b] ^ representing the empty site. Because we either have 
an occupied or an empty site, we need to implement the 
b] ^bi,^ + bl ^bi^l = 1 local constraint at every site. There- 
fore, we use d] = bl ^bi,i as the slave boson representation 
of the hardcore boson. It is clear from the definition of 
the bi along with the local constraint, that we either have 
It). = bl\0)^^ 4tl^a^)^' = |0) = 4jvac),, 

where |vac) is an unphysical state which does not host 
any quasiparticle (neither the empty nor the occupied 
state by the hardcore boson). 

Therefore, in the Schwinger boson approach, we de- 
compose the spin operator in terms of two fiavors of 
bosons Si = Ibjcrbi, where b^ = {bi^^,bi^i)^ . We also 

need to impose the local constraint: 6^ ^bf i + b^^ ^b^^i = 
1, to recover the physical Hilbert space. Using the 
Schwinger boson approach, we come to the following re- 
lations 
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^ 1 + jXs.ij' — xl^ijXs.i,j ^Iaj^sAJ (9) 

4S,.S, - -2A^ ,^^.At,,,, + 1 

= 1 + 2xl..jXt,»,, = xI,jXt,*j - Al,^,,,At,^j, (10) 

in which = b[^bj^^ + Xt.^.i = ^I,t^i,t - 

^a^j4. = Ktt'3,i-Kibj,^, and At,,j = &»,tfoj,; + 

bi,ibj^^. Therefore the second term in the Kane-Mele- 
Heisenberg Hamiltonian which is 17284. + J2Si.Sj de- 
couples as 

-^^2Al_,^^.A,,,,, - ^g2Al^JAt^^,J. (11) 

It is worthwhile mentioning that A^j^i = —Asaj, 
while Aj j i = At i j. Fa Wang in Ref.^^ has studied 
this model at 172 = using the Schwinger bosons. In the 



following we closely follow him and extend his study to 
the nonzero (72 values. To study the above model for- 
mally, we appeal to the Hubbard-Stratonovic transfor- 
mation followed by the saddle point approximation. Let 
us use the following definitions 

St,^,J = (At,.j) . (12) 

We also assume Xi,j = {x)ij — 0- To obtain the 
mean field Hamiltonian we assume the zero flux state 
pattern for Ai s and A2.S parameters which is introduced 
in Ref.'^*. We also assume a uniform s-wave A2.t in the 
mean field state. Accordingly, the mean field Hamilto- 
nian can be rewritten as follows 



H 



MF 









2,k ~Vk 



A* 



A2.k Vk \ 

-V*k ^2^k 

fi 

M / 



^-fc + Ns ^JiAi^s + 3J2A^_, + 3g2A 



2 

2,t 



(13) 



where Ns is the number of sites, ^'^ = 
(^'fe,A,&fc,i3,&Lfe,A'^-fe,B) ' and 

rjk = Y^i^s ^exp (-jfcy) + 2exp (ifcy/2) cos ^"^^^^^ 

A2,k = 6,fc + *Cs,fe, 

S,sM — 2 J2As,t sin cos {iky/2) — cos 

it,k = g2A2^f (^2 cos {Vikx/i^ cos {iky/2) + cos {yik;^^ 

The above Hamiltonian can be diagonalized using 
the Bogoliubov transformations which reduces to find- 
ing the eigenvalues of the M^A matrix where A = 
diag (1, 1, — 1, — 1). Accordingly, the energy dispersion 
has two branches as follows 



H'^ ±2 \r]k\ Cs,k 



\Vk\ -^Ik 



'^t,k- 



(15) 



Minimizing the total energy 



i^tot = Ns (-JiAls + 3J2Als + 352 Al, + ^k 



with respect to Ai_s, A2,s, and A2.t, we obtain the 
phase diagram of the KMH model. Among the self- 
consistency equations emerging from the minimization 
with respect to fj, we have the following constraint 



E 



-2 =0, 



(17) 



that implements the local constraint on the Hilbert space 
in average. To achieve the phase diagram we need 
to determine whether or not the energy excitation of 
Schwinger bosons i.e. E^ is gapped. If gapped we obtain 
the spin liquid phase, while the gapless case corresponds 
to the magnetic ordering. If bosons condense in site i 



condense such that {b. 



zi_i, and h. 



Z2 i , we have 



(14) 



(5,) (z) = {zl^z2,,) , 
{Sy) {i) = 3? {zI,Z2a) , 

{S.){^) = l(\zi\'-\Z2\' 



(18) 



We also need to find at which momentum Schwinger 
bosons condense. For example if we obtain A2^t = and 
Schwinger bosons condense at K — (0, 0) i.e. Eqq — 

0, we have /i — —77^,^(0,0) = The eigenvectors 

corresponding to the zero mode are given by (1, 0, 0, —1) 
and (0, 1, 1, 0). Assuming the weight of Shwinger bosons 
that condense at Eqq is zi and those at Eqq is Z2 we 
have 



(16) (*o,o)^ = {{bA,t) , {bB,t) , (bA^l)* , (bB^i)*) = (^i, ^2, ^2, -zi) (19) 



which means 
1 



{Sl)^^[\{bA^)\~\{bA,l)\ 

{Sh) = l{\{bB.t)f-\{bA,i)\' 
{S+) = {bA^^r (6^,;) = z*Z2, 



^0^ 
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FIG. 2. (Color online) Schematic mean field phase diagram of 
the KMH using the Schwinger model based on the numerical 
minimization of the total energy. 



(Si) = (6s,T>* M - -^2zl = - (5+) . 

Therefore the aforementioned condition leads to the 
Neel spin ordering. In the most general case, when 
bosons condense at k, the spatial profile of magnetiza- 
tion is a wave-packet with wave-vector k. That means 
(S)^ = exp (ik.Rji) {S)^ ^, where Rji = Rj — Ri and 
T — {A,B}. In case k is commensurate, i.e. k = ^Gi,2, 
where p, q are two integers with no common devisor and 
Gi,2 are basis vectors of the reciprocal lattice, the unit 
cell encloses 2\q\ sites. An interesting case is k = K and 
k = K', where K and K' are those momenta at which 
rik ~ 0. Since 3K — 3K' = Gi_2, the unit cell triples. 
Bosons condense at these momenta when J2 — >■ oo where 
we obtain two decoupled triangular lattices, and it is well 
known that magnetic ordering triples the unit cell and 
spins form 120 degrees with neighboring spins. It should 
be noted that the minimizing the total energy of bosons 
is somewhat tricky and there is small numerical error. 
Therefore, we have presented a schematic phase diagram 
in Fig. [2] based on our numerical results. 



is prone to exhibit magnetism of either commensurate 
or incommensurate Neel ordering forms. In this section, 
we are going to study the spin liquid phase more care- 
fully. To that end, we employ the Schwinger fermion 
approach to represent the spin operators. The decompo- 
sition procedure for the spin operators is the same as that 
of the Schwinger bosons expect that we replace bosonic 
operators with fermionic ones fi^a-- In the following, 
we consider the competition between spin liquids only 
as the starting point, i.e. we assume there is no long 
range magnetic ordering in the ground-state. However, 
after discussing the gauge theory of the KMH model, we 
argue that only the gapped spin liquid obtained in the 
previous section is stable against gauge fluctuations such 
as instanton effect for certain and other proposed spin 
liquid are likely unstable toward spontaneously broken 
symmetry phases. A key result is that the gapped spin 
liquid phase is a chiral spin liquid, i.e. the spin exci- 
tation in the bulk is gapped, while it has topologically 
protected gapless edge modes. This result is consist with 
several previous studies^^'^®'^^. 

The spin 1/2 operator can be represented in terms of 
Schwinger fermions with two flavors subject to a con- 
straint (such that the dimension of the local Hilbert space 
be two as that of the spin 1/2) as follows 



Sf = 



1. 



(21) 



Due to the constraint Sf operator can also be written 
as Sf — Ui^^ — 1/2 = 1/2 — rii^^. Using the Schwinger 
fermion redefinition of spin operator we can rewrite the 
Kane-Mele-Heisenberg Hamiltonian in the following way 

-•^11/2 E [x\{h])h{hf)+xl{i,3)xi{h]))m) 

in which we have defined Xa {h j) = flafj,'^ ^^'^ X ihj) = 
Xt (*i j) + Xl ihj)- The constraint can be implemented 
using the Lagrange multiplier method, though at half 
filling, it can be set to zero at the level of mean field ap- 
proximation. By implementing a Hubbard-Stratonovic 
transformation, the above Hamiltonian can be approxi- 
mated by the following effective Hamiltonian 



IV. SCHWINGER FERMION APPROACH 

As we discussed in the previous section, Schwinger bo- 
son approach is a useful tool to identify the spin liquid 
phase in the phase diagram. The mean field Schwinger 
boson method sparks the existence of the gapped spin 
liquid phase for the intermediate values of J2/J1 and for 
small values of 52 /J2- The rest of the phase diagram 



Hmf = - ^ Y ■^^Xi {hi) flafj,^ 

-\ E {j±X2,-a{l,jr +JllX2,-a{^,Jr)flJJA'23) 

where Xuihj) = (Xaihj))- To obtain the phase di- 
agram, we assume Xiihj) = Xi and X2,a{hj) = 
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X2 exp in which we have followed the conven- 

tion for used in the Kane-Mele-Hubbard model (see 
equation [1] for example). Using these assumptions the 



where 

Vk = — JiXi (^cxp {—iky) + 2 cos (^V3kx/2^ exp (iky /2)^2b) 

ik.cj = -X2 (-^2,11 singer + J2,±sin0-cr) Cfc 

Cfc = 2 sin [VZk^/2}j [cos {iky/2) - cos (y3k^/2^ ) (26) 

ik,a- -X2 (^2,11 COS (f>a- + J2,± COS cj)^„) (k 

Ck = 2 COS (^V3/ca;/2^ COS (3fcy/2) + cos (VSfc;^; j , (27) 
and the energy spectrum is given by 

Ek.. = 6,. ± vtF+el^- (28) 

At half filling all energy levels in the lower bands are 
occupied by Schwinger fermions. There are four param- 
eters in the total energy, xi, X2, 4>^ and (p^. To obtain 
their optimum values at half filling, the self consistency 

equations for Xa (i^j) — (^fi^afj.a^ should be solved. Al- 
ternatively, we can minimize the total energy with re- 
spect to those parameters and we obtain the same val- 
ues. We have solved the selfconsistency equations nu- 
merically and we obtain three different phases, 1- gap- 
less spin liquid^ 2- chiral gapped spin liquid, with nonzero 
Hall conductance and protected gapless edge states, and 
3- topological gapped spin liquid, with nonzero spin Hall 
conductance and protected gapless edge states (see Fig. 
[3] for the phase diagram). In the following we present 
more details on the nature of these phases. 

1- Gapless spin liquid phase. — Numerical min- 
imization of the total energy shows that when J2,\\ + 
\J2,±\ < l.TJi (or equivalently J2 < .85Ji and 52 < 
.85 Ji), the optimum value for X2, 4>t^ ^^^'^ 4>i Eire all equal 
to zero. In this phase, the energy dispersion of Schwinger 
fermions is identical to the energy dispersion of electrons 
in noninteracting graphene sheets with toff — JiXi- Ac- 
cordingly, the spin excitation (i.e. the excitation energy 
for Schwinger fermions) is gapless. On the other hand, it 
is easy to check that at the mean field level, there is no 
long range spin or charge ordering. Therefore this gap- 
less phase does not break any lattice symmetry and by 
definition is a gapless spin liquid phase. This result is 
however a consequence of the mean field approximation 
and needs to be more carefully studied under fluctua- 
tions around the mean fleld ground-state which is a re- 
sult of the strong correlations among Schwinger fermions. 



Hamiltonian becomes quadratic and is described by the 
following Matrix Hamiltonian for each spin degree of free- 
dom 



(24) 



r 




FIG. 3. (Color online) Mean field phase diagram of the KMH 
within the Schwinger fermion model. 



For example fluctuations of mean field parameters i.e. 
Xi,2 a-nd (p^^i may destroy the properties of the mean 
field state. It has been discussed in the literature that 
the most important fiuctuations which are should be in- 
cluded in any serious study are compact gauge fluctu- 
ations. In the next section we discuss that this state 
may undergo phase transition to the anti-ferromagnetic 
or VBS state28'29. 

2- Chiral gapped spin liquid phase. — For J2. + 

\J2,±\ > 1.7Ji and J2,_l > (or equivalently J2 > .85Ji 
and J2 > 32): the minimum of the ground-state energy 
manifold yields nonzero values for both xi and X2 and 
we obtain (p^ = (p^ — ±^ and the effective Hamiltonian 
for the Schwinger fermions within the mean field approx- 
imation is given by 

+i {J2,\\ + J2.±) X2 



fj- 1^ ft ft \ ( ^k,a + Cfe,cr Vk \ ( fk.A,a \ 
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which is identical to the Haldane model for the chiral 
state on the honeycomb lattice. Consequently, the mag- 
netic flux penetrating triangulares consisting of three 
neighboring same-sublattice sites equals $0- — ±30^ ^ 
for both and Schwinger fermions. The above model 
in the continuum model is described by two gapped Dirac 
cones around K ~ and K' = —K and the Chern of each 
band equals C"^ = sgn (c/)^) regardless of the flavor(spin) 
of the Schwinger fermions. Therefore the Hall conduc- 
tance of the system is ct|^ = C^"^^ = sgn (0^) j^. This 
means that the density of Schwinger fermions changes 
by AtIo- — sgn(0^) after inserting $ magnetic flux 
(which couples to Schwinger fermions). Since |C"^| = 1, 
there is one protected chiral edge mode for each spin 
degree of freedom. These chiral edge modes are ro- 
bust against disorder and their chirality is given by the 
sign of . Since X^o- ^ {0, tt}, the system breaks 
the time reversal symmetry. This can also be seen in 
the spin chirality value. Spin chirality is defined as 
Eij,/c = (Si.(Sj X Sk)) = 2j (xijXi./cXfc.i - Xi,feXfcjXi,i>: 
where Xi,j — So- fi afj,<^- ^P'^^ chirality operator is 
odd under the time reversal symmetry T, and parity P. 
Condensation of ^i,j,k implies the breaking of the time 
reversal and parity symmetries in the ground state. In 
this phase, it can be shown that Eij^k oc |$| when k 
belong to the same sublattice and are nearest neighbor 
of each other (belong to a triangle). However, within the 
meanfield level, the ground state respects the spin rota- 
tion, 6*6(120 degrees rotation symmetry), and the trans- 
lational symmetries. Therefore we name it chiral gapped 
spin liquid phase. 

3- Topological gapped spin liquid phase. — For 
^2.11 + \ J2,±\ > .85Ji and J2,_l < (or equivalently 92 > 
.85Ji and 172 > ■^2), we obtain both xi ^-nd X2 nonzero 
and = crt/) = icr^. The effective Hamiltonian for the 
Schwinger fermions within the mean field approximation 
is given by 

H =- JiXi E fU^'- 

(-^2.11 + J2,±) X2 

which is identical to the Kane-Mele model for the topo- 
logical insulator phase on the honeycomb lattice. Conse- 
quently, the magnetic flux penetrating triangulares con- 
sisting of three neighboring same-sublattice sites equals 
= ±30^ for and $^ = — for fi Schwinger 
fermions. The above model in the continuum model re- 
duces to two gapped Dirac cones around K — (1: 0) 

and K' = ~K and the Chern of each band equals 
C"^ — aC — asgn {cj)) depending on the spin of the 
Schwinger fermions. Therefore the Hall conductance 
of the system is cr|^^ = = (TSgn(0)j^. This 

means that the density of Schwinger fermions changes 
by Ario- — f7Sgn((/)) after inserting $ magnetic flux 
(which couples to Schwinger fermions). Therefore the to- 
tal number density of fermions i.e. An^ + An^ = 0, while 



the spin density i.e. 1/2 (An^ — An^) = sgn (0) is 
nonzero. Accordingly, the spin Hall conductance of the 
system is nonzero while fermions do not transfer across 
the edge of the system. Since = crC and |C| = 1, 
there is one protected chiral edge mode for each spin de- 
gree of freedom with opposite chiralities. The system 
does not break the time reversal(TR) symmetry and any 
TR preserving perturbation cannot destroy them due to 
Kramer's degeneracy. Therefore chiral edge modes are 
robust against non-magnetic disorder and their chirality 
is given by their spin and C . 



V. GAUGE THEORY OF THE 
KANE-MELE-HEISENBERG MODEL 

The Kane-Mele-Heisenberg model at half filling is de- 
scribed in terms of spin operators only. As we showed 
in section IV, these spin operators can be represented 
in terms of Schwinger fermions. It is straightforward to 
check that 5f, Sf, and Sf , are all invariant under the 
following local SU(2) gauge transformations'^^ 

fU^~Kh^ + <fU- (31) 

One way to see this is to consider the following matrix 

Spin operators can be described in terms of ipi in the 
following way 

S, = ^Tr (^J^.^T) , (33) 

where is the transpose of Pauli matrices, a. From the 
above definitions, it is obvious that spin operators under 
"04 — > hi^Ji where hi is a SU(2) unitary transformation. 
This transformation is equivalent to the transformation 
introduced in equation [32] provided hn = Ui and U12 — 
Pi. Beside the Hamiltonian, the action should also be 
invariant under SU(2) gauge transformations. To that 
end, we only need to show that the local constraints on 
the Hilbert space are also gauge invariant. SU(2) group 
has three generators, so there should be three constraints 
to be implemented through three temporal gauge fields 
that can serve as Lagrange multipliers. At half filling, at 
any site the total number of Schwinger fermions should 
by exactly one in order to retain the physical Hilbert 
space with two states per site. Therefore one constraint 
is fi^fi,t + fiifi,l = 1- Two other constraints at half 

filhng can be chosen as fiffj^ — fi,ifi,t = which are 
direct results of the first constraint. These constraints 
can be written as ipiipj = 1 or equivalently 
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Tr (^a^V,) = 0. (34) 

The above constraint is manifestly gauge dependent. 
These constraints can be implemented using Aq gauge 
fields. Therefore the Lagrangian is 

£ = ^Tr (^V4 + <^,< (*)) - H, (35) 

It is straightforward to check that the Lagrangian is 
invariant under the following simultaneous transforma- 
tions: Tpi — )■ hiij^i, and Aq (i) .cr — >■ hi [Aq (i) .a + i^] h\. 
This completes our claim of the SU(2) gauge invariance 
of the Kane-Mele-Heisenberg Hamiltonian. 

In the above paragraph, we showed that the original 
model has local SU(2) gauge symmetry. However, the 
meanfield solution of the system does not necessarily 
respect this property and can break the gauge symme- 
try down to U(l) or Z2 symmetries by the Anderson- 
Higgs mechanism. To determine the gauge theory of the 
meanfield state, we need to identify the invariant gauge 
group (IGG) of the meanfield solutions. To do so, we 

investigate the transformation properties of the (jl'i'>p]'^ 
matrices for every i and j sites, under the global SU(2) 
gauge transformations. When gauge particles are mass- 
less, these operators are invariant under any global gauge 
symmetries and the gauge fluctuations around the mean- 
field state is described by the original compact SU(2) 
gauge theory. When only a U(l) subgroup of the SU(2) 
gauge transformations, leaves all operators intact, the 
gauge theory is given by a compact U(l). Z2 spin liquid 
phase is also given by a meanfield ansatz that is invariant 
under a Z2 subgroup of the SU(2) gauge transformations 
only. It is easy to show that the IGG of the gapless spin 
liquid is SU{2) and the other two spin liquid phases have 
U(l) IGG and therefore we need to consider a spin liquid 
phase coupled to a compact SU(2) or U(l) gauge theories. 
Compact gauge theories are in principle hard to study as 
they include nonperturbative phenomena such as instan- 
ton effects (monopole configurations). Instanton (anti- 
instanton) change the value of the gauge potential by 
2tt (— 27r) and can potentially cause phase transition to 
symmetry breaking phases such as spin ordered phase, 
VBS or dimerized state and etc^^'^^. 



A. Gauge theory of the gapless spin liquid phase 

In this phase, the gapless Dirac fermions at K and K' 
points are coupled to a compact SU(2) gauge fiuctuations 
(QCD3). The spectrum of the gapless spin liquid phase is 
similar to the spectrum of the staggered flux phase. The 
instanton effect on the staggered flux has been exten- 
sively studied. Large N expansions indicates that QCDy, 
can be in the deconfined phase and therefore it might not 



undergo confinement transition'^^. Therefore the gapless 
spin liquid phase on the honeycomb lattice can be sta- 
ble against gauge fluctuations and might remain physical 
even after the instanton proliferation. It should be noted 
that it is not clear enough whether the large N stud- 
ies are applicable to the physical SU(2) case. If we be- 
lieve the other scenario where instantons destabilize the 
mean field state"^^, instanton condensation spontaneously 
breaks lattice symmetries^^'^^. For instance, we may ob- 
tain the VBS state or the Neel order as the ground-state 
of the KMH in this regime. 

B. Gauge theory of the chiral gapped spin liquid 
phase 

In the chiral gapped spin liquid phase, the mean- 
field state is described by the gapped Dirac fermions 
coupled to a compact U(l) gauge field. To obtain an 
effective action for the gauge field, we can integrate 
out fermions. Because of the gap in the spectrum of 
Schwinger fermions, we can easily obtain the Chern num- 
ber for their energy band. Doing so, we obtain the total 
Chern number equal to C = ±2 depending on the sign 
of (f)^. Therefore other than instantons, the U(l) gauge 
field contains a Chern-Simons terms in addition to the 
Maxwell action as follows 

^G.F. = j cPxdt ^e'^-^a^d.a^ - ^ {e^'^Pd^ax f (36) 

It is well-known that the Chern-Sinion term gaps out 
the gauge particles and therefore, we can neglect the 
gauge fiuctuations as well as the instanton effect in the 
low energy description. Accordingly, the chiral gapped 
spin liquid is stable against gauge fiuctuations and re- 
mains physical. 

We would like to mention that the ground state degen- 
eracy of the chiral spin liquid phase is four on the torus. 
To see why this happens to be true, we should notice 
that the chiral spin liquid breaks both time reversal and 
parity symmetries, so the degeneracy equals 2x2 = 4. 
Another way to check this is to note that fixing the sign 
of the spin chirality, e.g. Ei j f^ > 0, the ground state 
degeneracy on the 2D torus is given by the value of the 
Chern number. Since \C\ = 2, the ground state degen- 
eracy for each Eij^k > sector is two. Similarly the 
ground state degeneracy for the other sector is two as 
well. Therefore the total degeneracy is four on the torus. 
This topological invariant serves as good probe to iden- 
tify the spin liquid phase in numerical techniques such as 
quantum Monte Carlo (QMC) method. 

C. Gauge theory of the topological gapped spin 
liquid phase 

In the topological gapped spin liquid phase, Schwinger 
fermions are gapped in the bulk and the only low en- 
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FIG. 4. (Color online) Our proposal for the phase diagram of 
the KMH model. CSL stands for chiral spin liquid phase and 
AF denotes out of plane Neel ordering. XY ordering refers 
to the Neel order along an axis in the plane, e.g. along the x 
direction. 



ergy excitation is the compact gauge field. Since spin up 
Schwinger fermions and spin down Schwinger fermions 
have opposite Hall conductances, the total Chern num- 
ber vanishes as a result of the time reversal invariance of 
the ground state. Accordingly, the Chern-Simons action 
is absent in the low energy physics. Therefore, the gauge 
fluctuation is controlled by the gapless Maxwell term. 
However, instanton effect should also be taken into ac- 
count due to the compact nature of the gauge symmetry. 

In this phase, monopole insertion (instanton effect) 
adds a flux quantum of the gauge field to the system. 
Assuming — — — +1, a monopole insertion will 
increase the number of spin up Schwinger fermions by 
plus one and decreases the number of spin down by mi- 
nus one. Therefore, the flux quantum of the gauge field 
carries a charge of the spin operator Sz = 1 and flips the 
spin of Schwinger fermions. This means the density fluc- 
tuations of Sz generates the magnetic flux of the gauge 
field. Since the gauge fluctuation is gapless and linearly 
dispersed, it can be viewed as the Goldstone mode of the 
system. Consequently, there should be a spontaneous XY 
ordering in the system with a nonzero (^S^')^^. 



To conclude this section, at the mean field level we ob- 
tain a topological gapped spin liquid phase that does not 
break any lattice symmetry. The spectrum of fermions 
is gapped and so their fluctuations are suppressed. How- 
ever, the gauge fluctuation is still gapless and should be 
taken into account. Among those fluctuations, the most 
relevant one are the instanton effects. Instantons prolif- 
eration condensates and operators and as a result 
there is an in-plane (XY) spin ordering in the ground- 
state. Instantons also gap out the gauge field. Therefore 
we finally end up with a state that is not a spin liquid 
and instead it breaks the spin rotational symmetry spon- 
taneously. 

VI. SUMMARY AND DISCUSSION 

We have studied the strongly correlated limit of the 
Kane-Mele-Hubbard model. We derived the Kane-Mele- 
Heisenberg interaction as the effective model for the spin 
degree of freedom. Using the Schwinger boson approach 
we obtained the phase boundary between the spin liquid 
and the magnetically ordered phases. The spin liquid 
phase happens to be a gapped state. Therefore, both 
charge and spin degrees of freedom are gapped in the 
spin liquid phase. To have a better insight of the spin liq- 
uid nature, we studied the Kane-Mele-Heisenberg model 
through Schwinger fermion approach. Within the mean 
field theory, we identified three types of spin liquids. The 
gauge theory of the Schwinger fermion model was dis- 
cussed. We went beyond the mean field by taking the 
instanton effect into account. We discussed that instan- 
ton proliferation spoils all the spin liquid phases except 
the chiral spin liquid phase. However, it is not completely 
clear whether or not instantons will proliferate. Strong 
gauge fluctuations may generate pairing terms as well as 
hopping to farther neighbors. These terms can stabilize 
the spin liquid phase of matter. Moreover, large N limit 
studies of the compact SU(N) gauge fields yields the sta- 
bility of the spin liquid phase. Further investigation of 
the fate of the spin liquids is needed to determine what 
happens to the proposed topological as well as the gap- 
less spin liquid phases. Based on a combination of the 
Schwinger boson/fermion study, gauge theory and topo- 
logical arguments we suggest that the phase diagram of 
the KMH is as depicted in Fig. [4] . 
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